refactor: Rework Categorical/Enum to use (Frozen)Categories #23016
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
internal
orlp
force-pushed
the
cat-rework
branch
5 times, most recently
from
72307c2 to
863cf09
Compare
orlp
force-pushed
the
cat-rework
branch
4 times, most recently
from
ddb7532 to
9036ef6
Compare
This was referenced Jul 4, 2025
Member
Author
|
@coastalwhite I addressed most of your concerns, please respond to the others. |
2 tasks
2 tasks
2 tasks
7 tasks
Open
3 tasks
This was referenced Aug 7, 2025
1 task
2 tasks
2 tasks
2 tasks
2 tasks
2 tasks
2 tasks
dhimmel
added a commit
to dhimmel/openskistats
that referenced
this pull request
Oct 29, 2025
possibly related to pola-rs/polars#23016
lorentzenchr
added a commit
to lorentzenchr/model-diagnostics
that referenced
this pull request
Nov 11, 2025
* categorial/enum rework, see pola-rs/polars#23016
lorentzenchr
added a commit
to lorentzenchr/model-diagnostics
that referenced
this pull request
Nov 11, 2025
* categorial/enum rework, see pola-rs/polars#23016
This was referenced Nov 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
TLDR
Categoricals are completely reimplemented to be streaming compatible and fit better into the Polars Data model. They should generally be faster, more stable and more reliable. Physical ordering and the String Cache are gone. View #22568 for more context.
Fixes #3036.
Fixes #14247.
Fixes #14996.
Fixes #15293.
Fixes #15781.
Fixes #17479.
Fixes #17643.
Fixes #18065.
Fixes #18501.
Fixes #19868.
Fixes #19943.
Fixes #20290.
Fixes #20318.
Fixes #20364.
Fixes #20562.
Fixes #20878.
Fixes #20931.
Fixes #21175.
Fixes #21583.
Fixes #22448.
Fixes #22586.
Fixes #22664.
Fixes #22830.
Fixes #23015.
Fixes #23071.
Fixes #23289.
This PR, essentially, replaces the entire Categorical/Enum implementation. There is some breakage that was essentially unavoidable, unfortunately:
Categoricals has been removed, the ordering is now always lexical. The parameter has been deprecated, it is not a hard error to pass"physical"as ordering, it just doesn't do anything anymore.Enums in them are read back asCategoricals by older versions of Polars.Categoricaland integer types now always refer to the physical categories. These casts will be deprecated and removed at a later stage once we have dedicated functions to go to/from categories. The casts to/fromStringstill exist and will remain so, any other casts have been removed.The concept of local and global categories is gone. The
StringCachestill exists in Python, but does nothing anymore, and will be deprecated and removed later.In a future PR we will expose the new capabilities of the new
Categoriessystem, which lets you specify in the DataType which columns should share the same categorical mapping.